Using game responses to gather data

ABSTRACT

A system provides images or questions to multiple game participants and receives labels or answers in response thereto. The system uses the labels or answers for various data gathering purposes.

CROSS REFERENCE TO RELATED APPLICATION

The instant application claims priority from provisional application No.60/632,706, filed Dec. 3, 2004, the disclosure of which is incorporatedby reference herein in its entirety.

BACKGROUND

1. Field of the Invention

Implementations consistent with the principles of the invention relategenerally to data gathering and, more particularly, to using gameresponses to gather data.

2. Description of Related Art

Existing information searching systems use search queries to search datato retrieve specific information that corresponds to the received searchqueries. Such information searching systems may search informationstored locally, or in distributed locations. The World Wide Web (“web”)is one example of information stored in distributed locations. The webcontains a vast amount of information, but locating a desired portion ofthat information can be challenging. This problem is compounded becausethe amount of information on the web and the number of new usersinexperienced at web searching are growing rapidly.

Search engines attempt to return hyperlinks to web documents in which auser is interested. Generally, search engines base their determinationof the user's interest on search terms (called a search query) enteredby the user. The goal of the search engine is to provide links to highquality, relevant results to the user based on the search query.Typically, the search engine accomplishes this by matching the terms inthe search query to a corpus of pre-stored web documents. Web documentsthat contain the user's search terms are “hits” and are returned to theuser.

SUMMARY OF THE INVENTION

According to one aspect, a computer-implemented method may includeretrieving a first set of question-type queries from a query log havingknown answers and retrieving a second set of question-type queries fromthe query log having unknown answers. The method may further includeposing the first and second sets of queries to multiple users andreceiving the users' answers to the first and second sets of queries.The method may also include aggregating the answers from the users tothe second set of queries and determining correct answers for at leastsome of the queries of the second set of queries using the aggregatedanswers.

According to another aspect, a method may include providing a firstdigital image to multiple game participants and receiving first wordsfrom the multiple participants in response to the first digital image.The method may further include identifying the first words as relatedterms or synonyms.

According to a further aspect, a method may include eliciting userparticipation in a game hosted by a server and gathering data resultingfrom the user participation in the game. The method may further includeusing the gathered data for search related functions performed by asearch engine.

According to an additional aspect, a method may include providing adigital image to multiple game participants and associating a first wordin a first language with the digital image. The method may furtherinclude receiving a second word from a first one of the multipleparticipants in a second language in response to the digital image,where the second language is different than the first language. Themethod may also include identifying the second word as being a languagetranslation of the first word into the second language.

According to a further aspect, a method may include challenging a gameparticipant with multiple tasks, where a first portion of the tasks haveknown responses and a second portion of the tasks have unknownresponses. The method may also include verifying that the gameparticipant is human based on the participant's responses to the firstportion of the tasks and using the participant's responses to the secondportion of the tasks for purposes other than human verification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments of the invention and,together with the description, explain the invention. In the drawings,

FIG. 1 is a diagram of an exemplary network in which systems and methodsconsistent with principles of the invention may be implemented;

FIG. 2 is an exemplary diagram of a client and/or server of FIG. 1 in animplementation consistent with the principles of the invention;

FIG. 3 is a diagram of an overview of an exemplary quiz game used togather data to support question-type search queries in a search engineconsistent with an aspect of the invention;

FIGS. 4A, 4B and 4C are flowcharts of an exemplary process for analyzingparticipants' answers to quiz questions to determine correct answers forat least some of the quiz questions consistent with principles of theinvention;

FIG. 5 is a diagram of an exemplary quiz game show document consistentwith an aspect of the invention;

FIG. 6 is a diagram of an exemplary overview of the use of image labelsfor identifying the labels associated with an image as related terms;

FIGS. 7A and 7B are flowcharts of an exemplary process for providinglabels for images, and for identifying the image labels as relatedterms;

FIG. 8 is a diagram of an exemplary image label guessing documentconsistent with an aspect of the invention;

FIG. 9 is a diagram of an overview of a game challenge, according to anexemplary aspect of the invention, in which a game participant ischallenged with multiple tasks, and a portion of the participant'sresponses are used to verify that the participant is a humanparticipant, and another portion of the participant's responses are usedfor purposes other than human verification; and

FIG. 10 is a flowchart of an exemplary process, consistent withprinciples of the invention, for using participant responses to multipletasks to verify whether the participant is a human.

DETAILED DESCRIPTION

The following detailed description of the invention refers to theaccompanying drawings. The same reference numbers in different drawingsmay identify the same or similar elements. Also, the following detaileddescription does not limit the invention.

A “document,” as the term is used herein, is to be broadly interpretedto include any machine-readable and machine-storable work product. Adocument may include an e-mail, a web site, a file, one or more digitalimages, a combination of files, one or more files with embedded links toother files, a news group posting, a blog, a web advertisement, etc. Inthe context of the Internet, a common document is a web page. Web pagesoften include textual information and may include embedded information(such as meta information, images, hyperlinks, etc.) and/or embeddedinstructions (such as JavaScript, etc.). A “link” as the term is usedhere, is to be broadly interpreted to include any reference to or from adocument.

Exemplary Network Configuration

FIG. 1 is an exemplary diagram of a network 100 in which systems andmethods consistent with the principles of the invention may beimplemented. Network 100 may include multiple clients 110 connected tomultiple servers 120 and 115 via a network 130. Network 130 may includea local area network (LAN), a wide area network (WAN), a telephonenetwork, such as the Public Switched Telephone Network (PSTN), anintranet, the Internet, a memory device, or a combination of networks.Two clients 110 and two servers 120 and 115 have been illustrated asconnected to network 130 for simplicity. In practice, there may be moreor fewer clients and servers. Also, in some instances, a client mayperform the functions of a server and a server may perform the functionsof a client.

Clients 110 may include client entities. An entity may be defined as adevice, such as a wireless telephone, a personal computer, a personaldigital assistant (PDA), a laptop, or another type of computation orcommunication device, a thread or process running on one of thesedevices, and/or an object executable by one of these devices. Servers120 and 115 may include server entities that access, fetch, aggregate,process, search, and/or maintain documents in a manner consistent withthe principles of the invention. Clients 110 and servers 120 and 115 mayconnect to network 130 via wired, wireless, and/or optical connections.

In an implementation consistent with the principles of the invention,server 120 may include a search engine 125 usable by users at clients110. Server 120 may implement a data aggregation service by crawling acorpus of documents (e.g., web pages) hosted on data server(s) 115 andstore information associated with these documents in a repository ofcrawled documents. The data aggregation service may be implemented inother ways, such as by agreement with the operator(s) of data server(s)115 to distribute their hosted documents via the data aggregationservice. Search engine 125 may execute a query, received from a user, onthe corpus of documents hosted on data server(s) 115. Server 120 mayalso host various games, as described below with respect to FIGS. 3-10,to gather data for various purposes, such as, for example,search-related functions.

Server(s) 115 may store or maintain documents that may be crawled byserver 120. Such documents may include data related to published newsstories, products, images, user groups, geographic areas, or any othertype of data. For example, server(s) 115 may store or maintain newsstories from any type of news source, such as, for example, theWashington Post, the New York Times, Time magazine, or Newsweek. Asanother example, server(s) 115 may store or maintain data related tospecific product data, such as product data provided by one or moreproduct manufacturers. As yet another example, server(s) 115 may storeor maintain data related to other types of web documents, such as pagesof web sites.

While servers 120 and 115 are shown as separate entities, it may bepossible for one or more of servers 120 and 115 to perform one or moreof the functions of another one or more of servers 120 and 115. Forexample, it may be possible that two or more of servers 120 and 115 areimplemented as a single server. It may also be possible for a single oneof servers 120 or 115 to be implemented as two or more separate (andpossibly distributed) devices.

Exemplary Client/Server Architecture

FIG. 2 is an exemplary diagram of a client or server entity (hereinaftercalled “client/server entity”), which may correspond to one or more ofclients 110 and servers 120 and 115, according to an implementationconsistent with the principles of the invention. The client/serverentity may include a bus 210, a processing unit 220, an optional mainmemory 230, a read only memory (ROM) 240, a storage device 250, an inputdevice 260, an output device 270, and a communication interface 280. Bus210 may include a path that permits communication among the componentsof the client/server entity.

Processing unit 220 may include any type of software, firmware orhardware implemented processing device, such as a microprocessor, afield programmable gate array (FPGA), combinational logic, etc. Mainmemory 230 may include a random access memory (RAM) or another type ofdynamic storage device that stores information and instructions forexecution by processing unit 220, if processing unit 220 includes amicroprocessor. ROM 240 may include a conventional ROM device or anothertype of static storage device that stores static information and/orinstructions for use by processing unit 220. Storage device 250 mayinclude a magnetic and/or optical recording medium and its correspondingdrive.

Input device 260 may include a conventional mechanism that permits anoperator to input information to the client/server entity, such as akeyboard, a mouse, a pen, voice recognition and/or other biometricmechanisms, etc. Output device 270 may include a conventional mechanismthat outputs information to the operator, including a display, aprinter, a speaker, etc. Communication interface 280 may include anytransceiver-like mechanism that enables the client/server entity tocommunicate with other devices and/or systems. For example,communication interface 280 may include mechanisms for communicatingwith another device or system via a network, such as network 130.

As will be described in detail below, the client/server entity,consistent with the principles of the invention, may perform certaindata processing operations. The client/server entity may, in someimplementations, perform these operations in response to processing unit220 executing software instructions contained in a computer-readablemedium, such as memory 230. A computer-readable medium may be defined asone or more physical or logical memory devices and/or carrier waves.

The software instructions may be read into memory 230 from anothercomputer-readable medium, such as data storage device 250, or fromanother device via communication interface 280. The softwareinstructions contained in memory 230 may cause processing unit 220 toperform processes that will be described later. Alternatively, hardwiredcircuitry may be used in place of, or in combination with, softwareinstructions to implement processes consistent with the principles ofthe invention. Thus, implementations consistent with principles of theinvention are not limited to any specific combination of hardwarecircuitry and software.

Exemplary Quiz Game Overview

FIG. 3 illustrates an overview of a game that may be used to gather datato support question-type search queries. A log 305 of question-typesearch queries may be accumulated from multiple search queries of theform “what is xxx?,” “who is xxx?,” “where is xxx?,” or other types ofquestion formats, received at search engine 125. The question-typequeries may be accumulated from one or more users over a period of time.From the question-type query log 305, queries having known answers 310may be identified, and queries having unknown answers 315 may beidentified. For example, the question-type query “where is Akron?” mayhave the known answer “Ohio.” However, the question-type query “who isJoe Smith?” may have an unknown answer. In some implementations, querieshaving known answers 310 may be retrieved from any query source 370, andneed not be retrieved only from query log 305. The queries having knownanswers 310 and queries with unknown answers 315 may be issued to aparticipant 320 and a participant 325 from server 120. Participant 320may provide, to server 120, his answers 330 for the queries with knownanswers 310, and his answers 335 for the queries with unknown answers315. A score may be assigned 350, by server 120, to participant 320based on the participant's own answers 330 to the queries having knownanswers. For example, a score may be assigned based on a number of thequeries that participant 320 answered correctly.

Participant 325 may also provide, to server 120, his answers 340 for thequeries with known answers 310, and his answers 345 for the queries withunknown answers 315. A score may also be assigned 355, by server 120, toparticipant 325 based on the participant's answers 340 to the querieshaving known answers. A score, for example, may be assigned based on anumber of the queries that participant 325 answered correctly.

Participant 1 320's answers 335 to the queries with unknown answers 315,and participant 2 325's answers 345 to the queries with unknown answers315, may then be aggregated 360. The answers 335 and 345, for example,may be aggregated in a database associated with server 120. A correctanswer for each question-type search query may then be determined fromthe participants' aggregated answers 360. For example, an analysis ofthe answers 335 and 345 may determine that both participants (or amajority of participants, if more than two participants are involved)chose the same answer for a given question-type query and that,therefore, this answer is the correct answer for the query.

Exemplary Quiz Questioning Process

FIGS. 4A, 4B and 4C are flowcharts of an exemplary process, consistentwith principles of the invention, for analyzing participants' answers toquiz questions to determine correct answers for at least some of thequiz questions. As one skilled in the art will appreciate, the processexemplified by FIGS. 4A, 4B and 4C can be implemented in software andstored on a computer-readable memory, such as main memory 230, ROM 240or storage device 250 of server 120. In other implementations, theprocessing exemplified by FIGS. 4A, 4B and 4C can be implemented inhardwired circuitry, such as combinational logic, within processing unit220 of server 120.

The exemplary process may begin with the retrieval of question-typesearch queries from a query log (block 405) (FIG. 4A). The query log maybe accumulated from multiple search queries of the form “what is xxx?,”“who is xxx?,” “where is xxx?,” or other types of question formats,received at search engine 125 and stored in a database associated withserver 120. Queries from the retrieved question-type search queries thathave known answers may be identified (block 410). For example, a portionof the logged question-type search queries may have previously known,correct answers. Queries from the retrieved question-type search queriesthat do not have known answers may be identified (block 415). Forexample, a portion of the logged question-type search queries may nothave any known, correct answers associated with them. The determinationof questions with known or unknown answers may be made based on ananalysis of query logs and users' selections of search results. In someimplementations, queries having known answers may be retrieved from anysource, and need not be retrieved only from the query log.

A number of question-type search queries, from the retrieved querieswith known answers, may be issued to one or more participants (block420). For example, a set of search queries of the form “who is X_(i),”“what is X₂,” and “where is X₃” may be issued to the one or moreparticipants, where the correct answers to these questions are allknown. A number of question-type search queries, from the retrievedqueries having unknown answers, may be issued to the one or moreparticipants (block 425) (FIG. 4B). For example, a set of search queriesof the form “where is X₄,” “who is X₅,” and “what is X₆” may be issuedto the one or more participants, where the correct answers to thesequestions are not known. FIG. 5 illustrates the issuance ofquestion-type queries 505 to a participant via a game show document 510.

Answers, for each issued question-type search query, may be receivedfrom the participant(s) (block 430). As shown in FIG. 5, a participantmay enter his answers in answer fields 515 in game show document 510 andsubmit them via, for example, an answer submission 520 “button.” Inanother implementation, a document may present several candidate answersin a multiple choice format to a participant, and the participant maychoose his answer from the several candidate answers. A determinationmay then be made, for each participant, how many of the question-typesearch queries having known answers that the participant answeredcorrectly (block 435). A score may be assigned to each participant basedon the number of the question-type search queries, having known answers,that the participant answered correctly (block 440). For example, if agiven participant answered 90 of 100 question-type search queries havingknown answers correctly, then the participant may be assigned a score of90%. As shown in FIG. 5, a participant score document 525 may bepresented to the participant indicating the participant's assignedscore.

The answers received from the participant(s) may be aggregated (block445) (FIG. 4C) and stored in a memory, such as memory 230 of server 120,or in a database associated with server 120. A correct answer for eachof the question-type search queries having unknown answers may bedetermined using the aggregated answers (block 450). The answers fromthe participants, for the question-type search queries having unknownanswers, may be analyzed to ascertain which answer has been given themost for a specific question-type query. The answer that has been giventhe most by participants for a specific query may be determined to bethe correct answer. For example, if there have been 100 participants,and 10 of the participants selected the same answer, but everyone elseselected different answers, then the answer picked by the 10participants may not actually be the “correct” answer. A given answermay have to be selected a minimum number of times to be considered the“correct” answer. If each participant selected an answer from a multiplechoice list of candidate answers, then the answer that was chosen by amajority of participants may be determined to be the correct answer forthe question-type query. In one implementation, the answers given byparticipants that have been assigned a high score (see block 440 above)may be weighted more heavily when determining a correct answer for eachof the question-type search queries. Weighting high scoring participantsmore heavily when determining a correct answer for a search query mayremove the activity of “bots” (i.e., computer programs that runautomatically), since “bots” would likely not answer the question-typequeries correctly that have previously known answers. The determined“correct” answers to queries with previously unknown answers may be usedby a search engine for answering subsequently received question-typesearch queries.

Exemplary Image Labeling Game Overview

FIG. 6 illustrates an exemplary overview of the use of image labels foridentifying the labels associated with an image as related terms to oneanother. Multiple images (e.g., images 615-1 through 615-4) may bepresented to multiple participants (e.g., participant 605 andparticipant 610). The images may include, for example, digital images inany digital image format (e.g., MPEG, JPEG, etc.). For each image 615,each participant (e.g., participants 605 and 610) may supply a guessword (e.g., guess words 620-1 through 620-4 for participant 605 andguess words 625-1 through 625-4 for participant 610) that theparticipant believes describes the presented image. Each guess wordsupplied by a participant may then be identified as an image label(i.e., guess word 1_1 620-1 identified as label 1_1 630-1 for image 1615-1; guess word 2_2 625-2 identified as label 2_2 635-2, etc.). Theidentified image labels may then also be identified as related terms.For example, as shown in FIG. 6, label 1_1 and label 2_1, previouslysupplied as guess words from participant 605 and participant 610 forimage 1 615-1, may be identified as “related terms.” The labelsidentified as “related terms” may, for example, represent synonyms ofone another, or be otherwise related.

If the participants to the image labeling game are playing the game indifferent countries, the identification of image labels as “relatedterms” may be used for language translation also. For example, ifparticipants playing from France label the image of a car “voiture,” itcan be inferred that “voiture” is French for “car.” This technique maybe particularly useful in translating languages that are more obscurethan French.

Exemplary Image Labeling Process

FIGS. 7A and 7B are flowcharts of an exemplary process, according to animplementation consistent with the principles of the invention, forproviding labels for images, and for identifying the image labels asrelated terms. As one skilled in the art will appreciate, the processexemplified by FIGS. 7A and 7B can be implemented in software and storedon a computer-readable memory, such as main memory 230, ROM 240 orstorage device 250 of server 120. In other implementations, theprocessing exemplified by FIGS. 7A and 7B can be implemented inhardwired circuitry, such as combinational logic, within processing unit220 of server 120.

The exemplary process may begin with sending an image (image_i) to gameparticipants (block 705) (FIG. 7A). Multiple participants mayparticipate in each image labeling game. The image may be retrieved frommemory and sent from server 120 to each participant at a client 110.FIG. 8 illustrates an exemplary image label guessing document 805, thatis sent to each game participant, and that includes an image 810. Adetermination may be made whether a “pass” has been received from theparticipants (block 710), meaning that the participant does not have alabel for image 810. A participant may indicate a “pass” via, forexample, a “pass” button 815 provided in document 805 provided to theparticipant, as shown in FIG. 8. If a “pass” has been received from theparticipants, a determination may be made whether the image (image_i) isthe last image (block 715). Each game may only include a given sequenceof images, spanning a first image to a last image. If the image is notthe last image, then a counter i, used to designate a particular imagein an image sequence, may be incremented (i=i+1) (block 720). If theimage is the last image, then the exemplary process may continue atblock 765 below.

Returning to block 710, if a “pass” has not been received from theparticipants, then a guess may be received from each participant (block725). For example, each participant may type their next guess word infield 820 of document 805, as shown in FIG. 8. A determination may thenbe made whether any of the guesses is a “taboo” word (block 730). A“taboo” word may include a word that the game has excluded as anacceptable label for a given image. As shown in FIG. 8, document 805may, for each image 810, present a list 825 of “taboo” words to eachparticipant. If any of the guesses from the participants of the game isa “taboo” word, then an error message may be sent to the participant whosubmitted the “taboo” word (block 735), and the exemplary process mayreturn to block 725 above. If none of the guesses are “taboo” words,then a determination may be made whether the guesses of the participantsmatch (block 740) (FIG. 7B). For example, if there are two participantsto the game, then both participants may submit guess words that match.If the participants have not submitted matching guess words, then adetermination may be made whether a time period has expired (block 745).The time period may be configurable, and may represent an allottedamount of time that the participants to the game have to providematching labels for a given image. If the time period has not expired,then the exemplary process may return to block 725 above. If the timeperiod has expired, then the exemplary process may return to block 720above with the sending of a new image (image_i+1) to the gameparticipants.

Returning to block 740, if the guesses of the participants match, thenthe matching guesses may be stored as a label for the image (image_i)sent to the participants (block 750). A determination may be madewhether the image is the last image (block 755). If not, then theexemplary process may return to block 720 above. If the image is thelast image, then, in one implementation, labels associated with eachimage, from multiple games, or possibly multiple, differentparticipants, may be identified as related terms (block 760). Labels,associated with each image, may be aggregated from multiple differentgames and the labels may be identified as “related.” The image labelsmay be identified, for example, as synonyms. The labels (and theirsynonyms) may be used by a search engine for, for example, supplyingrelated search terms for a given input search term.

In another implementation, if the image is the last image, then a givenlabel from a participant in a first country in a first language may beidentified as a language translation of a corresponding label from aparticipant in a second country in a second language (block 765). Forexample, if one of the participants to the image labeling is playing thegame in the United States in English, and labels an image of anautomobile a “car,” and another participant is playing the game inFrance in French, and labels the image of the automobile “voiture,” itcan be inferred that “voiture” is French for “car.” “Voiture,” thus, maybe identified as the French translation of the English label “car.”

Exemplary Game Challenge Overview

FIG. 9 illustrates an exemplary overview of a game challenge, accordingto an exemplary aspect of the invention, in which a game participant ischallenged with multiple tasks, and a portion of the participant'sresponses are used to verify that the participant is a humanparticipant, and another portion of the participant's responses are usedfor purposes other than human verification. As shown in FIG. 9, aparticipant may be challenged with multiple tasks, with a portion 905 ofthe tasks being tasks that have known responses, and another portion 910of the tasks being tasks that have unknown responses 910. The tasks mayinclude any type of task, such as labeling an image, as described abovewith respect to FIGS. 6-8, or answering questions, as described abovewith respect to FIGS. 3-5.

The participant may provide a response 925 to each task 915-1 through915-x having known responses, and to each task 920-1 through 920-zhaving unknown responses. The participant may be verified 930 as beinghuman based on the responses 925 to each task having known responses.For example, if the task involves the participant providing answers toquestion-type queries, then the participant's responses 925 to tasks915-1 through 915-x can be compared to the known response to thosequestions to verify whether the participant is human, or whether theparticipant may possibly be a “bot”. A “bot” may not respond correctlyto the tasks to which the correct responses are known. The participantresponses 925 to the tasks 920-1 through 920-z having unknown responsesmay be used 935 for purposes other than human verification. For example,as described above with respect to FIGS. 3-5, participant answers toquestion-type queries that have unknown answers may be used to determinea correct answer to the question-type queries. As another example, asdescribed above with respect to FIGS. 6-8, image guess words provided bygame participants may used as related terms.

Exemplary Process

FIG. 10 is a flowchart of an exemplary process, according to animplementation consistent with the principles of the invention, forusing participant responses to multiple tasks to verify whether theparticipant is a human. As one skilled in the art will appreciate, theprocess exemplified by FIG. 10 can be implemented in software and storedon a computer-readable memory, such as main memory 230, ROM 240 orstorage device 250 of server 120. In other implementations, theprocessing exemplified by FIG. 10 can be implemented in hardwiredcircuitry, such as combinational logic, within processing unit 220 ofserver 120.

The exemplary process may begin with challenging a game participant withmultiple tasks (block 1005). Server 120 may send the multiple tasks tothe game participant at a client 110 via network 130. The multiple tasksmay include any type of task, such as issuing question-type queries to aparticipant, or presenting an image for the participant to label. Priorto the start of the game, correct responses to a first portion of themultiple tasks are identified, and correct responses to a second portionof the tasks are determined to be unknown. The game participant may thenbe verified as being human based on the participant's responses to thefirst portion of tasks (block 1010). The participant's responses to thetasks having known responses may be compared with the correct responsesto determine how many of the participant's responses are incorrect. Ahigh error rate may indicate that the participant is not human, and may,for example, be a “bot.” Responses identified as being potentially froma non-human source, such as a “bot” may be discarded. Responses from thegame participant to the second portion of tasks may then be used forother than human verification (block 1015). For example, as describedabove with respect to FIGS. 3-5, participant answers to question-typequeries that have unknown answers may be used to determine a correctanswer to the question-type queries. As another example, as describedabove with respect to FIGS. 6-8, image guess words provided by gameparticipants may used as related terms.

CONCLUSION

The foregoing description of preferred embodiments of the presentinvention provides illustration and description, but is not intended tobe exhaustive or to limit the invention to the precise form disclosed.Modifications and variations are possible in light of the aboveteachings or may be acquired from practice of the invention. Forexample, while series of acts have been described with regard to FIGS.4A, 4B, 4C, 7A, 7B and 10, the order of the acts may be modified inother implementations consistent with the principles of the invention.For the processes described with respect to FIGS. 4A, 4B, 4C, 7A, 7B and10, answers or responses from game participants may be solicited fromparticipants in many different ways than those described. In someimplementations, answers or responses may be solicited using freeforminput (e.g., if the correct answer/response is not known), or they maybe solicited using a multiple choice format (i.e., if a set of candidateanswers/responses is known that is believed to include a correctanswer/response). Additionally, participants may be incentivized to playthe games described above with respect to FIGS. 4A, 4B, 4C, 7A, 7B and10. Game participants may be rewarded (i.e., monetarily or otherwise)for participating in the games, and rewards may be used to leverage asignificant number of participants to participate in the games. In someimplementations, only a small subset of participants may be rewarded(e.g., a reward lottery), but the participants may not now who willreceive the reward until some time after participating in a given game.

It will also be apparent to one of ordinary skill in the art thataspects of the invention, as described above, may be implemented in manydifferent forms of software, firmware, and hardware in theimplementations illustrated in the figures. The actual software code orspecialized control hardware used to implement aspects consistent withthe principles of the invention is not limiting of the presentinvention. Thus, the operation and behavior of the aspects of theinvention were described without reference to the specific softwarecode—it being understood that one of ordinary skill in the art would beable to design software and control hardware to implement the aspectsbased on the description herein.

No element, act, or instruction used in the present application shouldbe construed as critical or essential to the invention unless explicitlydescribed as such. Also, as used herein, the article “a” is intended toinclude one or more items. Where only one item is intended, the term“one” or similar language is used. Further, the phrase “based on” isintended to mean “based, at least in part, on” unless explicitly statedotherwise.

1. A method performed by one or more processors associated with one ormore server devices, the method comprising: eliciting, by one or moreprocessors associated with the one or more server devices, userparticipation in a game by posing queries from a query log to a user,where the query log comprises search queries previously submitted by oneor more users; gathering, by one or more processors associated with theone or more server devices, first information resulting from the userparticipation in the game; gathering, by one or more processorsassociated with the one or more server devices, second information,which is different than the first information, resulting from the userparticipation in the game; assigning, by one or more processorsassociated with the one or more server devices, a score to the userbased on the gathered first information; assigning, by one or moreprocessors associated with the one or more server devices, a weight tothe gathered second information based on the assigned score; and using,by one or more processors associated with the one or more serverdevices, the weighted, gathered, second information to answer queriessubsequently posed to the one or more users.
 2. The method of claim 1,where the user participates in the game via a network connected to theone or more server devices.
 3. The method of claim 1, where elicitingthe user participation in the game comprises: retrieving, from the querylog, a first set of queries and retrieving answers associated with thefirst set of queries, retrieving, from the query log, a second set ofqueries that do not have associated answers, and posing, to the user, atleast one query from the first set and at least one query from thesecond set.
 4. The method of claim 3, where using the weighted,gathered, second information to answer queries, subsequently posed tothe one or more users, comprises: assigning, based on the score assignedto the user, the weight to the user's answer to the at least one queryfrom the second set; receiving the at least one query from the secondset, as a query from another user; selecting, based on the weight, theuser's answer to the at least one query from the second set; andproviding the selected answer to the other user in response to receivingthe at least one query from the second set, as a query from the otheruser.
 5. The method of claim 3, where assigning the score to the usercomprises: assigning, to the user, the score based on whether thereceived answer to the at least one query from the first set matched aretrieved answer associated with the at least one query from the firstset.
 6. The method of claim 1, where gathering the first informationcomprises: receiving answers, from the user, to the at least one queryfrom the first set; and gathering the second information comprises:receiving answers, from the user, to the at least one query from thesecond set.
 7. A computer-readable memory device comprising processinginstructions executable by one or more processors, the processinginstructions comprising: instructions for retrieving a first set ofqueries from a query log that comprises search queries previouslysubmitted by one or more users; instructions for retrieving answersassociated with the first set of queries from the query log;instructions for retrieving a second set of queries from the query logthat do not have associated answers in the query log; instructions forposing, to multiple users, at least one query from the first set ofqueries and at least one query from the second set of queries;instructions for receiving answers to the at least one query from thefirst set and the at least one query from the second set; instructionsfor aggregating the answers to the at least one query from the secondset; instructions for comparing answers to the at least one query fromthe first set to a retrieved answer associated with the at least onequery from the first set; instructions for assigning a score to eachparticular user, of the multiple users, based on a quantity of answers,received from the particular user, to queries from the first set ofqueries that match retrieved answers associated with the first set ofqueries; and instructions for selecting an answer for the at least onequery from the second set of queries based on the aggregated answers andscores assigned to the multiple users.
 8. The computer-readable memorydevice of claim 7, where the instructions for selecting the answercomprise: instructions for selecting an answer that was received from atleast a particular quantity of users.
 9. The computer-readable memorydevice of claim 7, where the queries are associated with images.
 10. Amethod performed by one or more processors of one or more computerdevices, comprising: retrieving, by at least one of the one or moreprocessors, a set of first queries from a query log that comprisessearch queries previously submitted by one or more users, where each ofthe first queries includes a question for which an answer has beenpreviously identified; retrieving, by at least one of the one or moreprocessors, a set of second queries from the query log, where each ofthe second queries includes a question that does not have an associatedanswer in the query log; posing, by at least one of the one or moreprocessors, a plurality of questions to a plurality of users, where theplurality of questions includes a plurality of the first queries and atleast one of the second queries; receiving, by at least one of the oneor more processors, answers to the plurality of questions from theplurality of users; determining, by at least one of the one or moreprocessors, a score reflecting an accuracy of each of the plurality ofusers based on the answers, provided by each of the plurality of users,that correspond to the first queries and that match the answerspreviously identified for the first queries; and selecting, by at leastone of the one or more processors, one of the answers for the at leastone of the second queries based on the answers provided by the pluralityof users and the scores for the plurality of users.
 11. The method ofclaim 9, where selecting one of the answers for the at least one of thesecond queries further comprises: selecting the one of the answers basedon (1) a weight given to each of the plurality of users, where theweight is based on the score associated with a respective user of theplurality of users, and (2) based on answers provided by a majority ofthe plurality of users.
 12. The method of claim 10, further comprising:verifying that a user, of the plurality of users, is human based on theanswers that correspond to the first queries.
 13. The method of claim10, further comprising: generating the answers to the first queriesbased on the answers that correspond to the second queries.
 14. A systemcomprising: at least one processor to: elicit user participation in agame by posing queries from a query log to a user, where the query logcomprises search queries previously submitted by one or more users;gather first information resulting from the user participation in thegame; gather second information, which is different than the firstinformation, resulting from the user participation in the game; assign ascore to the user based on the gathered first information; assign aweight to the gathered second information based on the assigned score;and answer queries subsequently posed to the one or more users based onthe weighted, gathered, second information.
 15. The system of claim 14,where the user participates in the game via a network connected to theone or more server devices.
 16. The system of claim 14, where, when theat least one processor is to elicit the user participation in the game,the at least one processor further is to: retrieve, from the query log,a first set of queries and retrieve answers associated with the firstset of queries, retrieve, from the query log, a second set of queriesthat do not have associated answers, and pose, to the user, at least onequery from the first set and at least one query from the second set. 17.The system of claim 16, where, when the at least one processor is toanswer the queries, subsequently posed to the one or more users, the atleast one processor further is to: assign, based on the score assignedto the user, the weight to the user's answer to the at least one queryfrom the second set; receive the at least one query from the second set,as a query from another user; select, based on the weight, the user'sanswer to the at least one query from the second set; and provide theselected answer to the other user in response to receiving the at leastone query from the second set, as a query from the other user.
 18. Thesystem of claim 16, where, when the at least one processor is to assignthe score to the user, the at least one processor further is to: assign,to the user, the score based on whether the received answer to the atleast one query from the first set matched a retrieved answer associatedwith the at least one query from the first set.
 19. The system of claim14, where, when the at least one processor is to gather the firstinformation, the at least one processor further is to: receive, from theuser, answers, to the at least one query from the first set; and whenthe at least one processor is to gather the second information, the atleast one processor further is to: receive, from the user, answers tothe at least one query from the second set.
 20. A system comprising: atleast one processor to: retrieve a first set of queries from a query logthat comprises search queries previously submitted by one or more users;retrieve answers associated with the first set of queries from the querylog; retrieve a second set of queries from the query log that do nothave associated answers in the query log; pose, to multiple users, atleast one query from the first set of queries and at least one queryfrom the second set of queries; receive answers to the at least onequery from the first set and the at least one query from the second set;aggregate the answers to the at least one query from the second set;compare answers to the at least one query from the first set to aretrieved answer associated with the at least one query from the firstset; assign a score to each particular user, of the multiple users,based on a quantity of answers, received from the particular user, toqueries from the first set of queries that match retrieved answersassociated with the first set of queries; and select an answer for theat least one query from the second set of queries based on theaggregated answers and scores assigned to the multiple users.
 21. Thesystem of claim 20, where, when the at least one processor is to selectthe answer, the at least one processor further is to: select an answerthat was received from at least a particular quantity of users.
 22. Thesystem of claim 20, where the queries are associated with images.
 23. Asystem comprising: at least one processor to: retrieve a set of firstqueries from a query log that comprises search queries previouslysubmitted by one or more users, where each of the first queries includesa question for which an answer has been previously identified; retrievea set of second queries from the query log, where each of the secondqueries includes a question that does not have an associated answer inthe query log; pose a plurality of questions to a plurality of users,where the plurality of questions includes a plurality of the firstqueries and at least one of the second queries; receive answers to theplurality of questions from the plurality of users; determine a scorereflecting an accuracy of each of the plurality of users based on theanswers, provided by each of the plurality of users, that correspond tothe first queries and that match the answers previously identified forthe first queries; and select one of the answers for the at least one ofthe second queries based on the answers provided by the plurality ofusers and the scores for the plurality of users.
 24. The system of claim23, where, when the at least one processor is to select the one of theanswers for the at least one of the second queries, the at least oneprocessor further is to: select the one of the answers based on (1) aweight given to each of the plurality of users, where the weight isbased on the score associated with a respective user of the plurality ofusers, and (2) based on answers provided by a majority of the pluralityof users.
 25. The system of claim 23, where the at least one processorfurther is to: verify that a user, of the plurality of users, is humanbased on the answers that correspond to the first queries.
 26. Thesystem of claim 23, where the at least one processor further is to:generate the answers to the first queries based on the answers thatcorrespond to the second queries.
 27. A non-transitory computer programproduct comprising: one or more programming instructions to: elicit userparticipation in a game by posing queries from a query log to a user,where the query log comprises search queries previously submitted by oneor more users; gather first information resulting from the userparticipation in the game; gather second information, which is differentthan the first information, resulting from the user participation in thegame; assign a score to the user based on the gathered firstinformation; assign a weight to the gathered second information based onthe assigned score; and answer queries subsequently posed to the one ormore users based on the weighted, gathered, second information.
 28. Thecomputer program product of claim 27, where the user participates in thegame via a network connected to the one or more server devices.
 29. Thecomputer program product of claim 27, where the one or more programminginstructions to elicit the user participation in the game furthercomprises one or more programming instructions to: retrieve, from thequery log, a first set of queries and retrieve answers associated withthe first set of queries, retrieve, from the query log, a second set ofqueries that do not have associated answers, and pose, to the user, atleast one query from the first set and at least one query from thesecond set.
 30. The computer program product of claim 29, where the oneor more programming instructions to answer the queries, subsequentlyposed to the one or more users, further comprises one or moreprogramming instructions to: assign, based on the score assigned to theuser, the weight to the user's answer to the at least one query from thesecond set; receive the at least one query from the second set, as aquery from another user; select, based on the weight, the user's answerto the at least one query from the second set; and provide the selectedanswer to the other user in response to receiving the at least one queryfrom the second set, as a query from the other user.
 31. The computerprogram product of claim 29, where the one or more programminginstructions to assign the score to the user further comprises one ormore programming instructions to: assign, to the user, the score basedon whether the received answer to the at least one query from the firstset matched a retrieved answer associated with the at least one queryfrom the first set.
 32. The computer program product of claim 27, wherethe one or more programming instructions to gather the first informationfurther comprises one or more programming instructions to: receive, fromthe user, answers to the at least one query from the first set; and theone or more programming instructions to gather the second informationfurther comprises one or more programming instructions to: receive, fromthe user, answers to the at least one query from the second set.
 33. Anon-transitory computer program product comprising: one or moreprogramming instructions to: retrieve a first set of queries from aquery log that comprises search queries previously submitted by one ormore users; retrieve answers associated with the first set of queriesfrom the query log; retrieve a second set of queries from the query logthat do not have associated answers in the query log; pose, to multipleusers, at least one query from the first set of queries and at least onequery from the second set of queries; receive answers to the at leastone query from the first set and the at least one query from the secondset; aggregate the answers to the at least one query from the secondset; compare answers to the at least one query from the first set to aretrieved answer associated with the at least one query from the firstset; assign a score to each particular user, of the multiple users,based on a quantity of answers, received from the particular user, toqueries from the first set of queries that match retrieved answersassociated with the first set of queries; and select an answer for theat least one query from the second set of queries based on theaggregated answers and scores assigned to the multiple users.
 34. Thecomputer program product of claim 33, where the one or more programminginstructions to select the answer further comprises one or moreprogramming instructions to: select an answer that was received from atleast a particular quantity of users.
 35. The computer program productof claim 33, where the queries are associated with images.
 36. Anon-transitory computer program product comprising: one or moreprogramming instructions to: retrieve a set of first queries from aquery log that comprises search queries previously submitted by one ormore users, where each of the first queries includes a question forwhich an answer has been previously identified; retrieve a set of secondqueries from the query log, where each of the second queries includes aquestion that does not have an associated answer in the query log; posea plurality of questions to a plurality of users, where the plurality ofquestions includes a plurality of the first queries and at least one ofthe second queries; receive answers to the plurality of questions fromthe plurality of users; determine a score reflecting an accuracy of eachof the plurality of users based on the answers, provided by each of theplurality of users, that correspond to the first queries and that matchthe answers previously identified for the first queries; and select oneof the answers for the at least one of the second queries based on theanswers provided by the plurality of users and the scores for theplurality of users.
 37. The computer program product of claim 36, wherethe one or more programming instructions to select the one of theanswers for the at least one of the second queries further comprises oneor more programming instructions to: select the one of the answers basedon (1) a weight given to each of the plurality of users, where theweight is based on the score associated with a respective user of theplurality of users, and (2) based on answers provided by a majority ofthe plurality of users.
 38. The computer program product of claim 36,further comprising one or more programming instructions to: verify thata user, of the plurality of users, is human based on the answers thatcorrespond to the first queries.
 39. The computer program product ofclaim 36, further comprising one or more programming instructions to:generate the answers to the first queries based on the answers thatcorrespond to the second queries.